Binary Classifier Metrics + ROC/PR + Threshold Curves

Positive prevalence p (P(Y=1)=p):

Decision threshold t (predict Ŷ=1 if score ≥ t):

Model separation (higher ⇒ easier problem)

Calibration noise (higher ⇒ noisier probabilities)

Random seed

Data generator (synthetic)

Labels Y ~ Bernoulli(p). Scores S in [0,1] from two Beta distributions: S|Y=1 ~ Beta(a₁, b₁) and S|Y=0 ~ Beta(a₀, b₀). Separation pushes class means apart; noise reduces Beta concentration.

Confusion matrix at threshold t

	Actual Y=1	Actual Y=0
Pred Ŷ=1
Pred Ŷ=0

Quick notes

Threshold curves (Sensitivity, Specificity, PPV vs threshold)

ROC curve (TPR vs FPR)

Precision–Recall curve (Precision vs Recall)

Binary Classifier Metrics Demo (N = 100,000): live metrics + ROC + PR + threshold curves